Abstract:
As a software engineer, I have been involved in software quality and testing of various forms of commercially developed software for overseven years. Many of these software applications ranging from database tierservers, LAN|Web communication, HR applications, installers, and printer devicedrivers. In all of these software examples, I developed various forms and levelsof automation to increase the efficiency of the testing effort with an equalvariety of success. It has been my experience that most of the successimplementing automated test tools has primarily not resided in the automationtools employed. Rather, it has existed more in the vision of the leadership tomake a paradigm shift from old methods of testing software in order to actualizethe value of automated test tools. Without an environment designed, accepted,and developed in all stages of the test planning process the success ofautomation can be plagued with delays, feature changes, and resistance tochange. The following testing strategy was developed to assist my HP colleaguesand partners to recognize the necessity to reduce the time and cost of testingmethods as it currently exists. I have borrowed from the HP Rules of the Garageas a template in defining the paradigm shift needed to facilitating better, faster, and smarter use of software testing resources.
Believe you can change the world.
Past success can be an invigorating reminder of how good a job we have done; that there is no need to alter the course of how we test ourproducts, or whether we need to investigate alternatives to the way we haveperformed testing on past products. In this view, past success can be adetriment to being able to change testing habits. To believe that one can changethe world requires that we are willing to make change ourselves. I think CarlyFiorino stated this best, "We must be willing to eat our own dog food�".This means you cannot simply advocate the benefits of automation. One mustactually implement and use the tools within a process improvement cycle � plan|do|check|act(the Shewhart cycle) (1 Grady). Automation development and testing mustbe understood as having no difference in application from that of the development practices involved in the software being tested.
Work quickly, keep the tools unlocked, and work whenever.
Although software testing is often constrained to follow the product development release schedule, it never-the-less should make every effortto optimize planning and arrange resources on its own well defined set of checkpoints. Setup time must be reduced to a minimum or completed prior to drop ofthe next code release. It is essential to develop processes that enable testingas soon as release drops occur. This requires significant reduction in setuptime associated with testing of the next software release cycle candidate. Inmost cases, there needs to be a consistent test environment established havingstatic test machines in place at all times. In addition, a table of objectivesand backup procedures should be prepared in advance to accommodate dynamictesting where a variety of SW|HW|OS configurations are necessary to complete thescope of planned testing. We cannot move quickly enough if we choose to wait until the next drop occurs.
Testing tools are only valuable if they get used. Particularly with complex automation, it is necessary for the tool users tobecome familiar with its use, provide for the expanded use of the tool, and mostimportantly, to provide usage feedback that further enhances the value of thetool in testing. It can then be determined if the support of the tool forrequired testing has merit or if a suitable alternative is available. Some ofthe risks in practice that can inherently devalue tool and automation investment are:
- Lack of a sponsorship authority that champions and continuously promotes use of the tool(s).
- Tools are implemented for testing that is ineffective, or not as originally intended.
- Support and involvement of partners are not incorporated in the tool design.
- Significant and sometimes un-communicated feature changes to the product being tested occur.
- Unclear ownership, expected outcome, and training in the use of the tool.
- Lack of motivation to use the tools; usage improvements thus cannot be determined rendering the tool obsolete or "locked".
When testing resources become scare, further improvements in efficiency can still be realized through automation that can be run unattendedor after-hours. "Lights out" testing has been developed and availablefor several years at HP. VCD has several automation tools built with Visual Test on site and leveraged from other sites.These tools have great value, have a history of proven use, and should continueto evolve to meet changing product testing demand. (Please find tool detail in Resources section of this document).
Know when to work alone and when to work together.
"Lasting productivity improvement must come from within and cannot be imposed from without." (5 Bumbarger).
If you want to work alone in an organization, propose radicalchange. Obviously this is not the time to work alone since progressive changerequires a receptive and collaborative effort. But the act of proposing changepresents many perceptual, habitual, cultural, emotional, and political obstaclesfrom those who would be expected participants and supporters. That is why theseenvironmental factors need to be addressed or it is futile to introduce thetools of change. Real, lasting productivity improvement requires change. Andchange requires creativity and innovation. The irony in productivity improvementis that success is most unlikely to succeed when imposed from without � but often is forced to change when ineffective from within.
Share tools, ideas. Trust your colleagues.
Results of using automation in software driver testing showed a reduction in testing time by half. There is also the inherent benefit ofconsistency of execution � repeatability. Thousands of keystrokes or settingscan be repeated exactly without the variation when done by hand. Automationreduces thrashing through defects that are not properly characterized and/orhard to duplicate. This results in thousands of reported defects over timeremaining open and without a clear resolution. Automation further benefits thetester by providing time savings when multiple SW|FW|HW|OS configurations arerequired. A multiple of tests can therefore be run concurrently and multiply the effectiveness of the testers time.
No politics. No bureaucracy.
Organizations by the very nature of their existence are political and as they grow - become bureaucratic. Our organization is noexception. But this does not mean that visionary thinking cannot be used toimprove or eliminate as many obstacles within the organization to effect positive change.
The customer defines a job well done.
From the standpoint of software quality, the customer as well as the next customer represents Argus project data, R&D partners, betatesting, outsource testing, and final media to name a few. Automation hasindirectly provided improvements in servicing the next customer via automateddefect handling tools to report defects more efficiently and earlier in thedevelopment cycle. Automated defect submittals have, for the past 2 years,contributed the largest share of reported defects at a savings of more than 771hours in submittal costs. We should continue to support these systems andprovide the technical resources and tools to maintain the reduced cost of defect handling.
